Shoppers presently can use their smartphones to scan product barcodes in stores, to obtain product information. This capability is expected soon to be available with headworn apparatus, such as Google Glass. Applicant's published patent application 20130223673, and IBM's published patent application 20140002643, detail how such product information may be presented using augmented reality (AR) techniques.
AR techniques commonly work by recognizing an input image, and identifying (and presenting) an augmentation that corresponds to that imagery. One application of AR is in promoting movies. A user is invited to capture an image of a movie poster (e.g., for the movie Transformers). A corresponding application on the user's smartphone recognizes the poster artwork, and overlays a rendering of a 3D character from the movie (e.g., a Transformer who breaks out of the poster and starts a rampage).
Use of AR in retail shopping is expected to be less thrilling but more useful. For example, such a capability will allow shoppers to obtain ingredient lists and discount information. Social network integration will enable presentation of information gleaned from the shopper's social network friends, including their personal product reviews.
Prior AR implementations have been based on 2D imagery, e.g., triggering the corresponding annotation, and placing it—with respect to features of the 2D artwork. But packaging is three dimensional. And product packaging often warrants more than a single annotation.
In accordance with one aspect of the technology, multiple differently-oriented views of a 3D retail product package (e.g., a cereal box) are captured during an enrollment phase. (A camera used to capture the images may be fixed, and the package may be positioned on a rotating turntable.) This collected imagery is then processed to identify landmarks on different faces of the product. The landmarks may be robust image features (e.g., SIFT points), or an invisible Cartesian coordinate system that is defined by an array of steganographic watermark tiles encoded in the package artwork. In some embodiments, the captured views are processed to yield a 3D model of the product.
This information is then used as a basis for authoring plural AR augmentations for the product. For example, a marketing designer at the company that produced the product (e.g., General Mills) may define different augmentations respectively linked to (1) the General Mills logo on the front of the box, (2) the Wheaties logo on the front of the box, (3) the athlete's picture (e.g., Michael Phelps) on the front of the box, (4) the Nutrition Facts panel on the side of the box, (5) a text blurb about Michael Phelps on the back of the box, etc. etc. The designer simply selects (e.g., by graphical highlighting) the region of the box that is to be associated with a particular augmentation, and provides a link to the desired action.
Software translates the designer-highlighted region into coordinates that can be machine-recognized (e.g., using SIFT feature points, or steganographic watermark data). Corresponding information identifying the region, and the designer-determined annotation that is to be associated with that region, are stored in a database.
When a consumer later captures imagery of the cereal box, the product is recognized, and a variety of different annotations are presented—depending on the viewpoint. Exploring the product—by physically moving the package, or by interacting with an on-screen user interface—reveals additional annotations.
The annotations needn't all be defined by the product producer. Some annotations may be defined by the store in which the product is sold (e.g., Safeway). Other annotations may be defined by consumers. A Darwinian process can proceed, in which the most favored annotations are those that are routinely presented.
The present technology thus enables product packaging to come alive with interactivity—each object a portal into a rich tapestry of diverse experiences.
The foregoing and additional features and advantages of the present technology will be more readily apparent from the following detailed description, which proceeds with reference to the accompanying drawings.